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Examiner's Amendment/Reasons for Allowance\ 

1 . An examiner's amendment to the record appears below. Should the changes 
and/or additions be unacceptable to applicant, an amendment may be filed as provided 
by 37 CFR 1.312. To ensure consideration of such an amendment, it MUST be 
submitted no later than the payment of the issue fee. 

In the Claims 

This listing of claims will replace all prior versions, and listings, of claims in the 
application. 

1. (Previously presented) A computer system computer application screen 
fingerprinter, said computer system comprising: 
a processor; 

a computer memory coupled to said processor; and 

a screen fingerprinter stored in said computer memory, wherein said fingerprinter 
comprises a decision tree that selects at least one region and/or pattern of screens of a 
presentation space of a computer application to be captured such that an occurrence of the at 
least one region and/or pattern enables the decision tree to uniquely identify each of the 
screens. 

2. (Original) A computer system as in claim 1 , wherein said fingerprinter allows 
a user to modify which portion of a screen comprises said region and/or pattern and which 
attributes of said region and/or pattern to examine. 

3. (Previously presented) A computer system as in claim 1, wherein said 
fingerprinter creates the decision tree based on said at least one region and/or pattern such 
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that after each screen is compared to the region or pattern at each decision node, a screen 
identifier will come to a different end node of said decision tree for each screen. 

4. (Original) A computer system as in claim 3, wherein said fingerprinter allows 
a user to modify said decision tree by modifying the comparisons at the decision nodes. 

5. (Previously presented) A computer system computer application recorder, 
said computer system comprising: a processor; a computer memory coupled to said 
processor; a user interface and a recorder stored in said computer memory, wherein said 
recorder records in said computer memory a knowledge base which comprises each screen of 
the presentation layer of a computer application, the keystrokes and/or programs necessary to 
reach each state, the available actions from each state of each screen and the effect of any 
actions available in each state through navigating said computer application in said user 
interface, 

6. (Original) A computer system as in claim 5, wherein said processor generates 
said file while a user navigates said another computer system in said user interface. 

7. (Original) A computer system as in claim 5, wherein said processor generates 
said file while automatically navigating said another computer system, 

8. (Previously presented) A computer system as in claim 5, wherein a 
fingerprint of each screen is included in said knowledge base. 
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9. (Previously presented) A computer system as in claim 5, wherein pre- 
conditions and postconditions for each state are included in said knowledge base! 

10. (Original) A computer system navigation planner, said computer system 
comprising; a processor; a computer memory coupled to said processor; at least one 
computer application model stored in said computer memory; and a navigation planner 
stored in said computer memory; wherein when said navigation planner receives a problem 
statement, said navigation planner accesses said at least one computer application model to 
create a plan of solving said problem statement and executes said plan. 

1 1 . (Original) A computer system as in claim 1 0, wherein when said plan fails, 
said navigation planner creates a new and different plan to solve said problem statement. 

12. (Previously presented) A computer system computer application model 
generator, said computer system comprising: 

a processor; 
a fingerprinter; 
a recorder; and 

v 

a user interface; 

wherein said fingerprinter selects at least one region and/or pattern of the screens of 
the presentation space of a computer application to be captured such that said at least one 
region and/or pattern of each screen is unique; 

wherein said recorder records in said computer memory a knowledge base which 
Comprises each screen of the presentation layer of a computer application, the keystrokes 



t 
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and/or programs necessary to each screen, a fingerprint of each screen, the available actions 
from each screen and the effect of any actions available in each screen through navigating 
said computer application in said user interface; 

wherein additional relationships between said screen and said knowledge base can be 
input through said user interface such that said computer application model generator can 
model said computer application. 

13. (Cancelled) 

14. (Original) A method of uniquely identifying the screens of the presentation 
layer of a computer application comprising the steps of: taking a screen capture of each 
screen of the presentation layer of a computer application; selecting areas of said screen 
captures to be examined for the presence of an attribute in said area; and creating a decision 
tree such that each of said screen captures has a unique end node of said decision tree. 

15. (Original) A method as in claim 14, wherein said areas are selected 
automatically. 

16. (Original) A method as in claim 14, wherein said areas are selected manually. 

17. (Original) A method as in claim 14, wherein said decision tree is created 
manually, 

18. (Currently amended) A method as in claim 14 further comprising: of 
reoording - feo - fltcitoo of a computer application comprising tho oteps of: accessing said ! 
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computer application; navigating said computer application; and recording in a knowledge 
base each screen of the presentation layer of said computer application, keystrokes and/or 
programs necessary to reach each state of each screen of said computer application, the states 
of each screen, available actions from each screen and (he effect of any actions taken on each 
screen. 

19. (Original) A method as in claim 18, wherein said computer application is 
navigated automatically. 

20. (Original) A method as in claim 18, wherein said computer application is 
navigated manually. 

21. (Currently amended) A method o fp l nnninc n aolutjonto a problem statement 
comprising! the stopa of: receiving a problem statement at a computer syste m, the problem 
statement specifying at least one goal ; accessing at leastone computer application model that 
encapsulates information on how at least one computer application operates: b controllod 
and/or data io-aocoogod: using a navigation p lanner m combination with the comp uter 
application model to plan pksBisg at least one path through said at least one computer 
application that will achieve the at least one goal of said problem statement; and executing 
said at least one path. 



1 
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22. (Previously presented) A method of modeling computer applications 
comprising the steps of: taking a screen capture of each screen of the presentation layer of a 
computer application; selecting areas of said screen captures to be examined for thepxesence 
of an attribute in said area; creating a decision tree such that each of said screen captures has 
a unique end node of said decision tree; accessing said computer application; navigating said 
computer application; and recording in a knowledge base each screen of the presentation 
layer of said computer application, keystrokes and/or programs necessary to reach each state 
of each screen of said computer application, the states of each screen, and the effect of any 
actions taken on each screen. 

23. (Previously presented) A method as in claim 22, further comprising the steps 
of: allowing a user to insert additional relationships and commands into said knowledge base. 

24. (Previously presented) A system for navigating an application comprising: 
a processor; 

a computer memory coupled to the processor; 

a screen fingerprinter stored in said computer memory, wherein said fingerprinter 
comprises a decision tree that selects at least one region and/or pattern of screens of the 
presentation space of a computer application to be captured such that an occurrence of the at 
least one region and/or pattern enables the decision tree to uniquely identify each of the 
screens; 

a recorder stored in the computer memory, the recorder recording a knowledge base 
which comprises the plurality of screen captures, one or more inputs and/or programs 
necessary to reach the application states indicated by each of the screen captures, one or more 1 
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actions available from each of the states, and the effects of undertaking each of the actions 
available from each of the states; and 

a navigation planner that receives a problem statement, creates a plan of solving the ■ 
problem statement by using the knowledge base, and executes the plan, 

wherein the creating of the plan by the navigation planner comprises the navigation 
planner dynamically analyzing a current screen to determine a current state, detemuning a 
desired state associated with the problem statement, and dynamically identifying a sequence 
of the actions from the current state need to achieve the desired state. 

2. Authorization for this examiner's amendment was fax transmitted on May 1 0, 
2007 from applicant's attorney, David M. O'Neill. 



Reasons for Allowance 

3. Claims 1-12, 14-24 are allowed. 

4. The following is an examiner's statement of reasons for allowance: 
The cited prior art taken alone or in combination fails to teach the claims 

invention of models that encapsulate information on how remote applications are 
controlled and data is accessed and created to be used by a runtime agent that 
intelligently reasons out goal-oriented plans and executes tasks to solve problem 
statements generated by a user interface or other software agents. Among other 
features, the process involves storing the capturing of sections of screen presentations 
for each other computer system representing a specific state, keystrokes that are 
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required to reach each state, information on how to recognize each screen in the 
presentation layer, the available actions from each state, and the related effect while 
navigating other computer systems in the user interface wherein such data is unique. 
An intelligent agent generates an optimized path to a desired end state of the other 
computer based on user inputs and displays the output in the intelligent agent's user 
interface. 

The closest prior art (Okerlund et al., USPN 6,690,371) teaches obtaining 
relevant image data extraction from a medical image data volume. Volume rendering 
(VR) is a preferred choice for visualizing a volume of medical image data. The VR 
process is governed by a set of curves, namely opacity and color/intensity curves. 
These curves determine what contribution and color intensity a given voxel (a data point 
within the volume) will have in the resultant 3D projection or rendering. The challenge is 
to determine an appropriate curve to accentuate important information (such as by 
providing contrast enhanced vessels, for example) for study while suppressing 
irrelevant information to the study (for example, scanner table and bone structures in a 
study of vessels). Volume clipping or cut planes also can be used to remove portions of 
a volume that are not of interest and that would otherwise obscure or occlude relevant 
tissue or anatomy. For example, volume clipping can be used to cut away an image of 
a rib to get an unobscured sagital view of an aorta. Okerlund teaches an interactive 
volume rendering. 

In relation to the independent claims of the invention, the applicant differentiates 
from Okerflund by definitions recorded in the specification at least in the following ways: 
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Claims 1,12,: A screen fingerprinter that captures sections of each screen of 
the presentation space from a computer system and stores the captured screen 
sections in a computer memory so that the captured section of each screen is unique 
(Specification, page 10, lines 3-7). Fingerprinting relies on matching pattern and region 
information to terminal screens, based on a decision tree. The patterns are special 
conditions that evaluate a terminal screen by looking for features, for example text 
strings, dates, times and character attributes such as color. Each feature can be 
searched for in several ways including exact position matches, region matches, 
exclusion, substring matches and so on. A pattern is captured and can be compared to 
a string to return a true or false for a match or can be evaluated more generally, for 
example, is there a data (regardless of what date) in a particular portion of the screen. 
The patterns can be evaluated as logical expressions. The values returned are used to 
navigate the decision tree (Specification, page 18, lines 14-22). 

Claim 5: The system includes a recorder to record in at least one file for each 
other computer system each state of the presentation layer of each other computer 
system each state of the presentation layer of each other computer system. In addition, 
the recorder stores those keystrokes that are needed to reach each state, how to 
recognize each screen in the presentation layer, the available actions from each state 
and the effect of any actions available in each state through navigating the other 
computer systems in the user interface (all computer systems are viewed from a 
common or one user intrerface). Recorder generates a model that can be used by an 
intelligent agent and such model can be embedded in an intelligent agent. As a result, 
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the processor generates an intelligent agent that includes a user interface 
(Specification, page 11, lines 4-14). The intelligent agent generates an optimized path 
to a desired end state of the other computer systems based on user inputs in the 
intelligent agent's user interface each time the intelligent agent is run. The intelligent 
agent executes the optimized path by accessing each other company system as 
necessary, retrieving output from the other computer systems and displaying the output 
in the intelligent agent's user interface (Specification, page 10, lines 8-13). 

Claims 10, 21 : A computer system navigation planner comprises a processor, a 
computer memory, and a user interface. A fingerprinter in the computer memory 
captures sections of each screen of the presentation space from at least one other 
computer system. The captured screen is stored in the computer memory in files for 
each other computer system so that the captured section of each screen in each of said 
files is unique (Specification, page 1 1 , lines 1-6). Above comments of Claim 1 apply. 

Claim 14: Above comments on fingerprinter apply. The decision tree used for the 
fingerprinting schema uses a region and pattern combination for each node. As the first 
node of the decision tree is traversed, the pattern for the node is evaluated; if the 
pattern is false, then the next node of the decision tree is traversed. If the pattern for 
the first node is true, then the region for the node is evaluated, and the result is stored 
as a fingerpring "element." The node of the tree that the screen matched and the 
element that was stored are the fingerprint of the screen (specification, page 19, lines 7- 
12). 

Claim 22: The above comments on fingerprinting and decision tree apply. 
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Claim 24: All of the above comments apply. 
Any comments considered necessary by applicant must be submitted no later than the 
payment of the issue fee and, to avoid processing delays, should preferably accompany 
the issue fee. Such submissions should be clearly labeled "Comments on Statement of 
Reasons for Allowance." 



Correspondence Information 

Any inquiry concerning this information or related to the subject disclosure 
should be directed to the Primary Examiner, Joseph P. Hirl, whose telephone number is 
(571 ) 272-3685. The Examiner can be reached on Monday - Thursday from 6:00 a.m. 
to 4:30 p.m. 

As detailed in MPEP 502.03, communications via Internet e-mail are at the 
discretion of the applicant. Without a written authorization by applicant recorded in the 
applicant's file, the USPTO will not respond via e-mail to any Internet correspondence 
which contains information subject to the confidentiality requirement as set forth in 35 
U.S.C. 122. A paper copy of such correspondence will be placed in the appropriate 
patent application. The following is an example authorization which may be used by the 
applicant: 

Notwithstanding the lack of security with Internet Communications, I hereby authorize the USPTO 
to communicate with me concerning any subject matter related to the instant application by e- 
mail. I understand that a copy of such communications related to formal submissions will be 
made of record in the applications file. 

If attempts to reach the Examiner by telephone are unsuccessful, the 
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Examiner's supervisor, David R. Vincent can be reached at (571 ) 272-3080. 
Any response to this office action should be mailed to: 

Commissioner of Patents and Trademarks, 

Washington, D. C. 20231; 
Hand delivered to: 

Receptionist, 

Customer Service Window, 
Randolph Building, 
401 Dulany Street, 
Alexandria, Virginia 2231 3, 

(located on the first floor of the south side of the Randolph Building); 
or faxed to: 

(571 ) 273-8300 (for formal communications intended for entry. 
Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov . Should 
you have any questions on access to Private PAIR system, contact the Electronic 
^utjT^^effer (EBC) at 866-217-9197 (toll free). 




^fsipfi P. Hirl 
Primary Examiner 
May 10, 2007 



